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DETAILED ACTION 



1 . This communication is responsive to an amendment filed on 9/27/04. 

2. Claims 1-3, 5-21 , 31 and 32 are pending in this application with claims 1,11 and 
17 being independent claims. Claims 1-3 and 5-21 have been amended; claims 4 and 
22-30 have been cancelled; and claims 31 and 32 have been added. This action is 
made Final. 

3. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 



Claim Rejections - 35 (JSC §112 

4. The following is a quotation of the second paragraph of 35 U.S.C. 1 1 2: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

5. Claims 9, 17-21 are rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

Claim 9 appears to depend on another claim; however, the claim it depends on is 
not disclosed. The examiner will continue to treat claim 9 as depending upon claim 8. 

Claim 17 recites the limitation "the device display" in lines 5, 7 and 15. There is 
insufficient antecedent basis for this limitation in the claim. The claim's preamble cites 
both "a device" and "a display". The examiner will interpret "the device display" to mean 
the device. Claim 17 also recites the limitation "the device display surface" in lines 9 and 
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12-13. There is insufficient antecedent basis for this limitation in the claim. The 
examiner will interpret "the device display surface" to mean the device's boundaries of 
control of the continuous display area/surface. Furthermore, the examiner is unclear 
what is meant by applicant's recitation of "a method for establishing a mapping between 
a virtual display space for a human user interacting [and] a device with a display 
comprising the steps of..." in claim 17 and will interpret the claim to mean a method for 
establishing a mapping between a human user's interaction with a device and a display 
within a virtual display space. 



Claim Rejections - 35 USC § 103 

6. Claims 1-3, 5, 6, 8, 10, 31 and 32 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Filo et al. ("Filo") in view of Rekimoto et al. ("Rekimoto"). 

As per claim 1 , Filo teaches a system for interacting with displays and devices 
that use such displays comprising: 

a display configured to interact with a part of the body of a user of the system 
such as the hand or the finger, or an ornament or an object or device worn on the body 
such as a glove or thimble (figs. 2(A-B), 4, 5(A-B) and 1 1 ); 

a camera adapted to have in its field of view at least one of the following: (i) the 
finger of the human user and pointing object, or (ii) the display and the reflection or 
effect the finger of the human user along with the pointing object produces on the 
display (col. 9, lines 30-38 and lines 49-52; col. 7, lines 19-22 and lines 51-54; col. 11, 
lines 32-55; col. 13, lines 34-35 and lines 10-13, 15-16; col. 14, lines 10-24); 
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a means for detecting the position of the finger of the human user or its reflection 
or effect on the display in an image registered by the camera (col. 7, lines 18-20; col. 
12, lines 26-34; col. 14, lines 10-19); and 

a means for establishing the mapping between the position of the finger of the 
human user or its reflection or effect on the display in the image registered by the 
camera and a corresponding location on the display (fig. 10; col. 18, line 52 through col. 
19, line 1; col. 7, lines 17-19; col. 12, lines 26-34; col. 14, lines 10-19). 

Although, Filo teaches a display configured to interact with a part of the body of a 
user of the system such as the hand or the finger, or an ornament or an object or device 
worn on the body such as a glove or thimble, Filo does not explicitly disclose the 
interaction between the user and display comprising a user's naked hand or finger and 
without additional attachments. Rekimoto teaches a system for interacting with displays 
and devices that use such displays wherein the interaction between the user and 
display comprises a user's naked hand or finger (figs. 1,7-9 and 14(A-B); col. 7, lines 5- 
1 1 ; col. 7, line 49 through col. 8, line 57; col. 9, lines 60-65; col. 1 0, lines 45-60; col. 1 5, 
line 60 through col. 16, line 36). Therefore, it would have been obvious to an artisan at 
the time of the invention to include Rekimoto's teaching of a system for interacting with 
displays and devices that use such displays wherein the interaction between the user 
and display comprises a user's naked hand or finger to Filo's teaching of a system for 
interacting with displays and devices that use such displays wherein the interaction 
between the user and display comprises a user's naked hand or finger with an 
attachment so that manipulation can be performed in a space in front of the display 
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utilizing direct manipulation without requiring the aid of a special pointing device and, 
consequently, provide a more advanced or enhanced interactive input/output 
environment for users of the system. 

As per claim 2, the modified Filo teaches a system for interacting with displays 
and devices that use such displays which commands the positioning of a pointing icon 
on the display (Filo: figs. 4B, 8, 10 ad 11; col. 11, lines 32-55; col. 12, lines 26-35; col. 
13, lines 34-35 and lines 10-13, 15-16; col. 9, lines 49-50). 

As per claim 3, the modified Filo teaches a system for interacting with displays 
and devices that use such displays wherein the system commands the input of data into 
the device using the display (Filo: fig. 10; col. 18, line 52 through col. 19, line 1; col. 7, 
lines 17-19; col. 12, lines 26-34; col. 14, lines 10-19). 

As per claim 5, the modified Filo teaches a system for interacting with displays 
and devices that use such displays wherein the pointing object is used to point to 
regions of the display by way of changing its position, attitude, or presentation (Filo: col. 
9, line 61 through col. 10, line 3). 

As per claim 6, the modified Filo teaches a system for interacting with displays 
and devices that use such displays wherein the pointing object is used to select, 
highlight, or define a particular point or region on the display (Filo: col. 9, line 61 through 
col. 10, line 3; col. 12, lines 26-35). 

As per claim 8, the modified Filo teaches a system for interacting with displays 
and devices that use such displays wherein the pointing icon on the display can be 
registered by the camera that has in its field of view the naked hand or finger of the 
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human user (Filo: figs. 2B, 8, 10 and 11; col. 11, lines 43-46; col. 12, lines 31-32; the 
sensor having camera capabilities, i.e. by definition, a camera is the part of a television 
transmitting apparatus that receives the primary image and transforms it into electrical 
impulses] Rekimoto: figs. 14(A-B) and respective portions of the specification), Filo does 
not explicitly disclose the pointing icon being registered by the camera. Hansen teaches 
a pointing icon being registered by a camera (col. 4, lines 27-31; col. 7, lines 18-27; col. 
8, lines 40-45; col. 9, lines 3-25). 

As per claim 10, the modified Filo teaches a system for interacting with displays 
and devices which also includes at least one of the following (Filo: col. 9, line 49 through 
col. 10, line 22; col. 7, lines 40-49; col. 8, lines 10-21, 30-37; col. 11, lines 61-63; col. 
13, lines 17-24; col. 14, lines 26-32): 

a) a method for selecting or highlighting a specific item or icon on the display; 

b) a method for activating a specific process, program, or menu item represented 
on the display; and 

c) a method for writing, scribing, drawing, highlighting, annotating, or otherwise 
producing marks on the display. 

As per claim 31 , the modified Filo teaches a system for interacting with displays 
and devices that use such displays comprising another camera and wherein the - 
pointing icon on the display can be registered by the other camera (Rekimoto: fig. 12, 
elements 4A and 4B). 

As per claim 32, the modified Filo teaches a system for interacting with displays 
and devices that use such displays wherein the means for detecting only detects the 
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position of the naked hand or finger of the human user in an image registered by the 
camera without a display background (Rekimoto: col. 16, lines 5-17). 
7. Claims 11-16 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hansen in view of Rekimoto et al. ("Rekimoto"). 

As per claim 1 1 , Hansen teaches a method for detecting a position of a pointing 
object or its reflection or effect on a display in an image registered by a camera 
comprising the steps of: 

a) defining at least one characteristic of the pointing object or its reflection or 
effect on the display that (i) is registered in the image captured by the camera and (ii) 
distinguishes the pointing object from other objects registered in the image from the 
camera (col. 3, lines 28-40; col. 8, lines 30-49), 

b) retrieving the image from the camera (col. 8, lines 12-20), 

c) analyzing the image from the camera to locate the characteristic or 
characteristics of the pointing object or its reflection or effect on the display pointing 
object (col. 8, lines 12-50), 

d) determining the most likely position of the pointing object or its reflection or 
effect on the display in the image from the camera based on at least one of the 
following: (i) the last known position of the pointing object or its reflection or effect on the 
display in the image, (ii) the position or positions at which the at least one distinguishing 
characteristic of the pointing object, or its reflection or effect on the display or the set of 
the picture elements in the image that comprise the rendition of the pointing object, or 
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its reflection or effect on the display (col. 4, lines 27-31 ; col. 7, lines 18-27; col. 8, lines 
40-45; col. 9, lines 3-25). 

Although, Hansen teaches a display configured to interact with a part of the body 
of a user of the system such as the hand or the finger along with the pointing object, 
Hansen does not explicitly disclose the interaction between the user and display 
comprising a user's naked hand or finger. Rekimoto teaches a system for interacting 
with displays and devices that use such displays wherein the interaction between the 
user and display comprises a user's naked hand or finger (figs. 1,7-9 and 14(A-B); col. 
7, lines 5-1 1 ; col. 7, line 49 through col. 8, line 57; col. 9, lines 60-65; col. 1 0, lines 45- 
60; col. 15, line 60 through col. 16, line 36). Therefore, it would have been obvious to an 
artisan at the time of the invention to include Rekimoto's teaching of a system for 
interacting with displays and devices that use such displays wherein the interaction 
between the user and display comprises a user's naked hand or finger to Hansen's 
teaching of a system for interacting with displays and devices that use such displays 
wherein the interaction between the user and display comprises a user's naked hand or 
finger along with a pointing object so that manipulation can be performed in a space in 
front of the display utilizing direct manipulation without requiring the aid of a special 
pointing device and, consequently, provide a more advanced or enhanced interactive 
input/output environment for users of the system. 

As per claim 12, the modified Hansen teaches a method for detecting a position 
of a naked hand or finger of a human user in an image registered by a camera wherein 
at least one characteristic that distinguishes the pointing object/naked hand or finger 
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from other objects in the image registered by the camera is known a priori (Hansen: col. 
4, lines 55-56; Rekimoto: figs. 1,7-9 and 14(A-B); col. 15, line 60 through col. 16, line 
36). 

As per claims 13 and 14, the modified Hansen teaches a method for detesting a 
position of a naked hand or finger of a human user in an image registered by a camera 
wherein at least one characteristic that distinguishes the pointing object/naked hand or 
finger from other objects in the image from the camera is determined based on analysis 
of at least one image of the same pointing object/naked hand or finger acquired from 
the camera and wherein at least one characteristic that distinguishes the pointing 
object/naked human hand or finger from other objects, whose rendition are present in 
the image from the camera is obtained by acquiring at least two images from the 
camera, one with the pointing object in view of the camera and one without, and 
comparing the two sets with one another (Hansen: col. 4, lines 1 5-41 ; col. 7, lines 1 8- 
27; col. 8, lines 38-45; Rekimoto: figs. 1,7-9 and 14(A-B); col. 7, lines 5-11; col. 7, line 
49 through col. 8, line 57; col. 9, lines 60-65; col. 10, lines 45-60; col. 15, line 60 through 
col. 16, line 36). 

As per claim 15, the modified Hansen teaches a method for detecting a position 
of a naked hand or finger of a human user in an image registered by a camera wherein 
adjustments or modifications are made to the position, viewing angles, sensitivity, and 
other settings of the camera pursuant the analysis of the data or image retrieved from 
the camera (Hansen: col. 4, lines 12-18). 
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As per claim 16, the modified Hansen teaches a method for detecting a position 
of a naked hand or finger of a human user in an image registered by a camera wherein 
at least part of the procedures for the method is carried out using at least in part the 
computing mechanism available on one or more of the following: the display, or the 
camera, or the pointing device, or the device producing the signal shown on the display, 
or the device producing the pointing icon on the display (Rekimoto: figs. 1,7-9 and 14(A- 
B); col. 7, lines 5-11; col. 7, line 49 through col. 8, line 57; col. 9, lines 60-65; col. 10, 
lines 45-60; col. 15, line 60 through col. 16, line 36). 

8. Claims 17-21 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hansen in view of Filo, and further in view of Rekimoto et al. ("Rekimoto"). 

As per claim 17, Hansen teaches a method for establishing a mapping between a 
human user's interaction with a device and a display within a display space comprising 
the steps of defining the boundaries of the positions that the pointing object can assume 
in addressing points or regions on the device and defining, within the boundaries, a 
continuous display surface (Abstract; col. 3, lines 21-42; col. 3, line 63 through col. 4, 
line 8; described is a display space for human user interaction with a device and display 
wherein user's manipulation of the set of positions that a pointing object assumes in 
addressing a set of corresponding points or regions on the display comprises the steps 
of defining the boundaries of the positions that the pointing object can assume in 
addressing points or regions on the display device and defining the boundaries of the 
display space), defining the boundaries of the device and defining within the boundaries 
of the display a continuous display surface (Abstract; col. 3, lines 21-42; col. 3, line 63 
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21-42; col. 3, line 63 through col. 4, line 8; defined are a continuous display surface and 
the device's boundaries of control of the continuous display area/surface), warping the 
geometry of the continuous display surface so that at least one region of the continuous 
display surface overlaps with at least one region of the device's boundaries of control of 
the continuous display area/surface and establishing a one-to-one or many to one 
correspondence between the overlapping regions of the device's boundaries of control 
of the continuous display area/surface and the continuous display surface (col. 7, lines 
1 3-20; the geometry of the pointing object surface is scaled or warped so that regions of 
the pointing object surface have an area in common with/overlaps with a region of the 
display surface to establish a one-to-one or many to one correspondence between 
regions of the pointing object surface and display surface). 

However, Hansen does not explicitly disclose segmenting the continuous display 
surface and pointing object surface into at least two regions in a virtual display space. 
Filo teaches a method for establishing a mapping between a human user's interaction 
with a device and a display within a virtual display space comprising segmenting a 
continuous display surface and associated/corresponding device's boundaries of control 
of the continuous display area/surface into at least two regions (figs. 2B, 8, 10 and 11; 
col. 3, lines 38-42; col. 11, lines 43-46; col. 12, lines 26-35; col. 14, lines 10-24; col. 18, 
line 52 through col. 19, line 5; segments of the display corresponds with segments of 
pointing object surface selected and wherein detecting a display perspective is inherent 
when there is already a method for detecting the pointing device in an interactive virtual 
world environment) and warping the geometry of the continuous display surface so that 
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at least one region of the continuous display surface overlaps with at least one region of 
the device's boundaries of control of the continuous display area/surface and 
establishing a one-to-one or many to one correspondence between the overlapping 
regions of the device's boundaries of control of the continuous display area/surface and 
the continuous display surface (col. 17, line 33 through col. 18, line 24; col. 18, line 52 
through col. 19, line 5). Therefore, it would have been obvious to an artisan at the time 
of the invention to include Filo's method for establishing a mapping between the set of 
positions that a pointing object can assume in addressing a set of corresponding points 
or regions on the display comprising segmenting the display surface and pointing object 
surface into at least two regions in a virtual display space to Hansen's method for 
establishing a mapping between the set of positions that a pointing object can assume 
in addressing a set of corresponding points or regions on the display in order to make 
the work environment more like a real life work environment as well as provide users 
with the ability to compartmentalize or organize areas of focus according to content. 

Hansen and Filo still do not explicitly disclose the interaction between the user 
and display comprising a user's naked hand or finger. Rekimoto teaches a system for 
interacting with displays and devices that use such displays wherein the interaction 
between the user and display comprises a user's naked hand or finger (figs. 1,7-9 and 
14(A-B); col. 7, lines 5-1 1 ; col. 7, line 49 through col. 8, line 57; col. 9, lines 60-65; col. 
10, lines 45-60; col. 15, line 60 through col. 16, line 36). Therefore, it would have been 
obvious to an artisan at the time of the invention to include Rekimoto's teaching of a 
system for interacting with displays and devices that use such displays wherein the 
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interaction between the user and display comprises a user's naked hand or finger to 
Hansen and Filo's teaching of a system for interacting with displays and devices that 
use such displays wherein the interaction between the user and display comprises a 
user's naked hand or finger along with a pointing object so that manipulation can be 
performed in a space in front of the display utilizing direct manipulation without requiring 
the aid of a special pointing device and, consequently, provide a more advanced or 
enhanced interactive input/output environment for users of the system. 

As per claim 18, the modified Hansen teaches a method for establishing a 
mapping between a human user's interaction with a device and a display within a virtual 
display space wherein the boundaries of the set of positions that a pointing object/naked 
hand and finger can assume are obtained by positioning a camera in a way such that 
the camera has in its view the naked hand or finger of the human user and wherein the 
boundaries of the set of positions that the pointing object/naked hand and finger can 
assume are obtained by querying the user to point to the boundaries (Hansen: Abstract; 
col. 3, lines 21-42; col. 3, line 63 through col. 4, line 8; Filo: col. 12, lines 5-22; col. 12, 
lines 26-35; col. 14, lines 10-24; Rekimoto: figs. 1,7-9 and 14(A-B); col. 15, line 60 
through col. 16, line 36). 

As per claim 19, the modified Hansen teaches a method for establishing a 
mapping between a human user's interaction with a device and a display within a virtual 
display space wherein the boundaries of the set of positions that a naked hand or finger 
can assume are obtained by positioning a camera in a way such that the camera has in 
its field of view the human user, locating the human user in the image from the camera 
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and deducing the positions the hand or the finger of the user can assume from the 
position of the human user in the image from the camera (Rekimoto: figs. 1,7-9 and 
14(A-B); col. 7, lines 5-1 1 ; col. 7, line 49 through col. 8, line 57; col. 9, lines 60-65; col. 
10, lines 45-60; col. 15, line 60 through col. 16, line 36). 

As per claim 20, the modified Hansen teaches a method for establishing a 
mapping between a human user's interaction with a device and a display within a virtual 
display space wherein the regions of the virtual display surface comprise at least two 
sets of pixel elements that comprise the image on the display (Hansen: col. 5, lines 1- 
15; Filo: col. 12, lines 5-22). 

As per claim 21, the modified Hansen teaches a method for establishing a 
mapping between a human user's interaction with a device and a display within a virtual 
display space wherein at least part of the procedures for the method is carried out using 
at least in part the computing mechanism available on one or more of the following: the 
display, or the camera, or the device producing the signal shown on the display, or the 
device producing the pointing icon on the display (Hansen: fig. 1; col. 2, line 66 through 
col. 3, line 55; Filo: figs. 1 and 2; col. 6, lines 6-66; Rekimoto: figs. 1,7-9 and 14(A-B); 
col. 7, lines 5-11; col. 7, line 49 through col. 8, line 57; col. 9, lines 60-65; col. 10, lines 
45-60; col. 15, line 60 through col. 16, line 36). 

9. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Filo et al. 
("Filo") in view of Rekimoto et al. ("Rekimoto") as applied to claim 1 , and further in view 
of Edwards et al. ("Edwards"). 
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As per claim 7, the modified Filo teaches a system for interacting with displays 
and devices that use such displays comprising a pointing object/naked hand or finger 
for entering information into the system and can be used in conjunction with various 
software applications (Filo: col. 7, lines 46-54; col. 8, lines 16-35; col. 10, lines 24-31 
and line 57; col. 13, lines 21-25; information entered into the system further include a 
clipboard on which a user can draw; Rekimoto: figs. 1,7-9 and 14(A-B); col. 7, lines 5- 
11; col. 7, line 49 through col. 8, line 57; col. 9, lines 60-65; col. 10, lines 45-60; col. 15, 
line 60 through col. 16, line 36). The modified Filo does not explicitly disclose the 
pointing object/naked hand or finger to be used to define a vector on the plane of the 
display that indicates a direction and magnitude relative to or with respect to an item on 
the display or a region of the display. Edwards teaches a system for interacting with 
displays wherein the pointing object to be used to define a vector on the plane of the 
display that indicates a direction and magnitude relative to or with respect to an item on 
the display or a region of the display (col. 8, lines 1 2-21 ). Therefore, it would have been 
obvious to an artisan at the time of the invention to include Edwards' teaching of a 
computer aided design tool, wherein the pointing object to be used to define a vector on 
the plane of the display that indicates a direction and magnitude relative to or with 
respect to an item on the display or a region of the display, to the modified File's 
teaching of a system for interacting with displays that includes drawing capabilities to 
provide users with an environment capable of incorporating multiple applications and 
capabilities to enhance a user's individual task. 
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10. Claim 9 is rejected under 35 U.S.C. 103(a) as being unpatentable over Filo et al. 
("Filo", US 6,215,498 B1) in view of Rekimoto et al. ("Rekimoto") as applied to claim 8, 
and further in view of Applicant's Admitted prior art. 

As per claim 9, although the modified Filo teaches a system for interacting with 
displays and devices that use such displays comprising means for sensing the naked 
finger/pointing object's position relative to the position of a pointer icon on the display 
(Filo: col. 7, lines 1-19; Rekimoto: figs. 1,7-9 and 14(A-B); col. 7, lines 5-11; col. 7, line 
49 through col. 8, line 57; col. 9, lines 60-65; col. 10, lines 45-60; col. 15, line 60 through 
col. 16, line 36), the modified Filo does not explicitly disclose the system to include a 
means for correcting the offsets between the position of the naked finger/pointing 
object, or reflection, or effect thereof on the display as observed by the user or by the 
camera, and the position of the pointer icon on the display. However, Applicant's 
admitted prior art teaches a system for interacting with displays and all devices that use 
such displays to include a means for correcting the offsets between the position of the 
pointing object, or reflection, or effect thereof on the display as observed by the user or 
by the sensor or the camera, and the position of the pointer icon on the display (page 
21 , lines 6-9). Therefore, it would have been obvious to an artisan at the time of the 
invention to include Applicant admitted prior art's teaching of a system for correcting the 
offsets between the position of the pointing object and the position of the pointer icon on 
the display to the modified Filo's system for sensing the naked finger/pointing object's 
position relative to the position of a pointer icon on the display in order to reduce the 
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margin of error concerning the position of the pointing object and the position of the 
pointer icon on the display. 

Response to Arguments 

1 1 . Applicant's arguments with respect to claims 1 -3, 6 and 10-16 have been 
considered but are moot in view of the new ground(s) of rejection. 

Conclusion 

12. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Trajkovic (US 6,531,999) teaches a pointing direction calibration in video 
conferencing and other camera-based system applications. 

Omura et al. (US 6,421,042 B1) teach a coordinate position inputting/detecting 
device, a method for inputting/detecting the coordinate position and a display board 
system. 

Pryor et al. (US 6,720,949 B1 ) teach man machine interfaces and applications. 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 
706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 
1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 



Application/Control Number: 09/757,930 Page 18 

Art Unit: 2174 

TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 



Inquires 

13. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Examiner Le Nguyen whose telephone number is (571) 
272-4068. The examiner can normally be reached on Monday - Friday from 7:00 am to 
3:30 pm (EST). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kristine Kincaid, can be reached on (703) 308-0640. 

The fax numbers for the organization where this application or proceeding is 
assigned are as follows: 

(703) 872-9306 [Official Communication] 
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Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

KSSST^X tGEOAJD 

lvn suf^.:-.^v:-t;-^h 

Patent Examiner 1^^..^— ^ --'2^0 

February 27, 2005 



